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Abstract 


The two purposes of this paper are to provide a SAS IML macro that performs loglinear 
smoothing and to apply this macro to loglinear smoothing problems that have not been 
extensively discussed in the test-equating literature. The SAS macro is demonstrated on 
univariate, bivariate, and trivariate smoothing problems. The univariate and bivariate examples 
reproduce published results (von Davier, Holland, & Thayer, 2004). The trivariate example 
extends the bivariate smoothing example to allow for comparisons of subgroups’ univariate and 
bivariate distributions. The implications are that important questions about distribution 
differences and subpopulation invariance of equating functions can be considered through 
comparisons and evaluations of complex loglinear models that are easily fit with this SAS IML 
macro. 
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Introduction 


Polynomial loglinear models for one-, two- and higher-way contingency tables (Bock & 
Yates, 1973; Haberman, 1974a, 1978, 1979) have important applications to measurement and 
assessment (Hanson, 1991; Holland & Thayer, 1987, 2000; Rosenbaum & Thayer, 1987). Two 
such applications are test score distribution estimation and comparison (Kolen, 1991; Hanson, 
1996). Another application is the estimation and enhancement of test-equating stability (von 
Davier, Holland, & Thayer, 2004; Holland & Thayer, 1989; Kolen & Brennan, 1995; Livingston, 
1993; Skaggs, 2004). In these applications, the polynomial loglinear models are essentially 
regarded as a smoothing technique that is commonly referred to as loglinear smoothing. 

In an effort to make loglinear smoothing more readily available, reports have described 
how it can be implemented with SAS/STAT PROC GENMOD (Moses & von Davier, 2004; 
Moses, von Davier, & Casabianca, 2004;SAS Institute, 2002a). PROC GENMOD is flexible and 
adequate for most simple univariate smoothing problems. However, it can have convergence 
problems for some bivariate loglinear smoothing problems. Moreover, PROC GENMOD does 
not directly provide the so-called “C-matrices”—that is, the low-rank matrix factors of the 
covariance matrix of the estimated probabilities (von Davier et ah, 2004; Holland & Thayer, 
1989, 2000) that are important computational tools for the standard errors of the smoothed 
frequencies and the accuracy measures used in the kernel equating framework. 

The possibility of developing a SAS IML (SAS Institute, 2002b) macro that implements 
loglinear smoothing without the limitations of PROC GENMOD was investigated. The purpose 
of this paper is to describe this new SAS macro (rather than to demonstrate PROC GENMOD) 
and to apply it to problems that have not been extensively discussed in the literature. The SAS 
macro performs loglinear smoothing according to Holland and Thayer’s (1987, 2000) 
specifications. It is appropriate for univariate, bivariate, and trivariate frequency distributions of 
test data, and it converges even when PROC GENMOD fails. This macro also computes the C- 
matrix factors. 

The first major section of this paper reviews the use of loglinear models for smoothing 
discrete distributions. The second section describes how to obtain and use the SAS macro to fit 
loglinear smoothing models. The third and fourth sections demonstrate the macro with respect to 
a simple univariate smoothing problem and a much more complicated 22-parameter bivariate 
problem, both from von Davier et al. (2004). The fifth section demonstrates the SAS macro on a 
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trivariate smoothing problem, where the third variable defines a subgroup that provides a basis 
for comparing bivariate distributions (see also Liou, Cheng, & Li, 2001). The implications of 
these applications are discussed in terms of future research with a focus on the study of equating 
methods. 


Loglinear Smoothing Models 

Assume we have a random variable X that defines the test fonn X (we use the same 
notation for a test form and a random variable) with possible values xo,...,xj, or xj, with j = 

the possible score values) and a corresponding vector of observed score frequencies n = 

(ino,...,nj) 1 that sum to the total sample size N. Under some distributional assumptions about n, 
like multinomial or Poisson distributional assumptions, the vector of the population score 
probabilities p = (po, is said to satisfy a loglinear model if 

\og e (Pj) = a + u j + b$ ( 1 ) 

where the {pj) are assumed to be positive and sum to one, bj is a row vector of constants 
referred to as score functions throughout this text (e.g., Xj xj Xj ), p is a vector of free 
parameters, u, is a known constant that specifies the distribution of the {pj} when P = 0, and a is 
a normalizing constant that ensures that the probabilities sum to one. 

Under different choices of u, B (the matrix of score functions formed by arranging the 
row vectors, bj, one on top of the other), or /?, the loglinear model becomes equivalent to the 
discrete uniform distribution (u = 0, P = 0) or the binomial distribution (see Holland & Thayer, 
1987, 2000, for details). 

Loglinear models are a class of exponential families of discrete distributions, which can 
be described in terms of their sample moments. As in Holland and Thayer (1987, 2000), we will 
make use of this property and of the fact that the w. are known constants. Therefore in this paper 

the loglinear model used to fit a univariate distribution is 

log, ip j ) = oc + Yj P, i x j )'» ( 2 ) 

i=l 

where the u } are set to zero. When the data are test score data, the terms in this model can be 
defined as follows: the xj are score functions of the possible score values of test A (e.g., xj _ x/. 
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Xj Xj ), a is as described above, and the /3 j are free parameters to be estimated in the model¬ 
fitting process. 

The value of / determines the number of moments of the actual test score distribution that 
are preserved in the smoothed distribution. If / = 1 then the smoothed distribution preserves the 
first moment (the mean) of the observed distribution. If / = 4 then the smoothed distribution 
preserves the first, second, third, and fourth moments (mean, variance, skewness, and kurtosis) 
of the observed distribution. 

The model in (2) can be extended to fit the bivariate distribution of the scores of two tests 
(call them X and Y)\ 

log,(^ i ) = a + ZA,(^)' + t^n i )*+t i/V*,)*(v») / , (3) 

/=1 h =1 g=l /=1 

where p jk is the joint score probability of the score (x h va; score x, on test X and score va on test 

Y). The fitting of (3) produces a smoothed bivariate distribution that preserves / moments in the 
marginal (univariate) distribution of X; H moments in the marginal (univariate) distribution of Y; 
and a number of cross-moments (G < /, F < H) in the bivariate X-Y distribution. Model (3) is 
also appropriate for the smoothing of bivariate distributions with impossible X- Y score 
combinations, structural zeros, when the total test score can never be less than the score on the 
internal anchor test and the anchor score cannot be less than its maximum possible value to a 
greater extent than the total test score is less than its maximum possible value (see Holland & 
Thayer for an example, 2000). 

Indicator functions can be used to fit both the full univariate distribution and a subset of 
the distribution (e.g. teeth or lumps at different score points) within a single loglinear model. One 
example of such a model is: 

l°g e (Pj ) = a + P x {Xj )‘ + f3 2 (Xj f + /? 3 Sj + /y (Xj ySj, (4) 

where the indicator function S'. = 1 if j belongs to a defined subset of all j’s and S'. = 0 
otherwise. S'. denotes the set of score points where the frequencies are systematically lower or 
higher than most of the test frequencies. Model (4) will preserve the mean and variance of the 
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total distribution of X (/?/ and /??), the total frequency in the cells denoted as S. = 1 (/%), and the 

mean of the cell values for the cells in S'. = 1 (fh). 

One additional smoothing model combines the bivariate model in (3) with the use of 
indicator functions in (4): 

log, (Pjki )=a+M+z a, ( y +x a* O'* )*+z xa/ ( x j y (y* y 

i =1 *=1 g=l /=1 ^ 

+£&«(.*,)'s, + + £ £/} sls (Xj)‘(y,ys, 

i =1 *=1 g=l /=1 

The model in (5) is useful for preserving features in a trivariate distribution, where p jkl is the 

probability of score (x h jy) in subgroup S/ = 0 or 1. Model (5) preserves the subgroups’ 
frequencies, X and Y univariate moments, and XY cross-moments. Simpler versions of (5) can 
include fewer subgroup-varying terms and can allow the subgroups’ distributions to share certain 
parameters. For example, less J3 xis tenns can be included, allowing for a certain number of the 

lowest univariate moments in X to vary by subgroups, but constraining the higher moments to be 
equal so that they are shared by the subgroups and equal to those of the total distribution. 

Fitting Loglinear Smoothing Models 

Under the assumption that the vector of the frequencies is multinomial, the estimation of 
the free parameters (/?,) proceeds by maximizing the following log-likelihood function: 


z= zv°g«(^)’ 

j 

where rij and Pj are the observed frequencies and the population score probabilities in the jth 

cell, respectively (Holland & Thayer, 1987, 2000). 

The maximization of (6) can be accomplished through the use of the Newton-Raphson 
algorithm (Holland & Thayer, 1987, p. 11). Holland and Thayer specify two criteria for the 
convergence solution from the algorithm. One criterion involves the maximization of the log- 
likelihood function; the maximum is said to be attained when the relative change in the log- 
likelihood is less than some specified value. The second criterion involves the satisfaction of the 
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likelihood equation for all of the estimated parameters (P), meaning that the relative error in each 
fitted moment must be less than some specified value. At convergence both criteria should be met. 

To add stability to the Newton-Raphson algorithm, the score functions in B are 
transformed so that they sum to zero and their squares sum to one (Holland & Thayer, 1987, 

2000; Rosenbaum & Thayer, 1987). Holland and Thayer also suggest specific starting values. 

The suggested starting values for the parameter estimates are based on converting the observed 
frequencies into a smoother form with nonzero frequencies at ah score points and then 
computing a function of these converted frequencies and the score functions. 

Large-sample standard errors of the estimated parameters (P) can be estimated when the 
Newton-Raphson algorithm converges to a maximum likelihood solution. The parameter 
estimates and standard errors that correspond to the higher moments are misleadingly small 
because they are coefficients of scores raised to high powers. If the comparability of parameter 
estimates is of interest, a preferable approach to defining score functions in terms of powers 
would be to define them as orthogonal polynomials (Haberman, 1974a). 

At convergence, the variance-covariance matrix of p is given as 

CovCp) = ( B'Cov{n)BY 1 , (7) 

where Cov(n ) = N(D - pp ) and D p is the diagonal matrix of p . 


Evaluating the Fit of Loglinear Models 

There are several measures that are useful for evaluating the extent to which the 
smoothed frequencies match the observed frequencies. The likelihood ratio chi-square statistic is 
given as: 


G 2 


2 H n A 0 


PjN 


( 8 ) 


where p j is the smoothed value of pj based on a particular model. This measure is often used in 

statistical tests that evaluate the relative fit of nested and competing models (Agresti, 2002; 
Haberman, 1974b; Hanson, 1996; Holland & Thayer, 2000). 

Other measures for overall model fit include the Pearson chi-square statistic, 
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(9) 


(nj - pjNf 

j Pj N ’ 

the Freeman-Tukey chi-square statistic, 

<io) 

j 

and two measures that penalize the overfitting of data, including the Akaike information criterion 
(AIC; Akaike, 1981, 1987), which adds twice the number of parameters estimated by the model 
to the likelihood chi-square statistic, and the consistent Akaike infonnation criterion (CAIC; 
Bozdogan, 1987), which adds l+log(A0 times the number of parameters to the likelihood ratio 
chi-square statistic. Variants of these measures not computed with the SAS IML macro include 
other members of Cressie and Read’s power-divergence family of chi-square statistics (Read & 
Cressie, 1988), the Bayesian inference criterion (Schwarz, 1978), and Gilula and Habennan’s 
modification of the AIC (Gilula & Haberman, 1994). 

In addition to evaluating overall model fit, it can be useful to compare the smoothed and 
observed frequencies at each score level using Freeman-Tukey residuals (Freeman & Tukey, 
1950), 

+ \ l n j + 1 - V 4 Pj N + 1 • (u) 

When a model fits the data well, the Freeman-Tukey residuals are asymptotically normally and 
randomly distributed with a mean of zero and a variance approaching one. The asymptotic 
variance of the Freeman-Tukey residuals is less than one and the departure from one depends on 
the complexity of the model and the sparseness of the data (Bishop, Feinberg, & Holland, 1975 
Haberman, 1973, 1974b;). Freeman-Tukey residuals are especially useful for suggesting whether 
indicator functions or higher moments are warranted in univariate distributions. The residuals 
become less useful when there are many zeros in the observed frequencies (as in many bivariate 
problems). 


C-Matrices 

The estimated variance-covariance matrix of the smoothed probabilities (2^) can be used 
for obtaining their confidence intervals (Holland & Thayer, 1987, 2000) and for computing 
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standard errors of kernel equating (von Davier et al., 2004). A very useful factorization of the 
estimated variance-covariance matrix is the C-matrix, defined as: 




( 12 ) 


where C ' is the / by / matrix that can be efficiently computed as: 


C p =N 


- 1/2 


D r P Q 


(13) 


The diagonal matrix, D , has the diagonal entries 
that comes from the following ()/?-factorization: 



, and Q is the / by / orthogonal matrix 



jjp 1 B = QR. 


(14) 


Q is a J by I matrix with orthogonal columns, R is an / by / upper triangular matrix, and B is the 
matrix of score functions and shown in (1) (Holland & Thayer, 1987). The QR call routine in SAS 
returns a /by / matrix (SAS Institute, 2002b), so the SAS IML macro uses the first / columns of 
the outputted Q in computing the C-matrix (Dongarra, Bunch, Moler, & Stewart, 1979). 


A SAS Macro for Loglinear Smoothing 

The SAS macro described in this paper is flexible enough to address several of the 
loglinear smoothing problems described in the literature, including univariate, bivariate, and 
trivariate problems, and provides all of the fit measures reviewed in the previous section and the 
C-matrix. The requirements for implementing the macro are: SAS software, SAS IML, and some 
familiarity with SAS DATA and PROC statements. This macro will be distributed upon request 
by the first author. The 6-step procedure for implementing the macro within SAS is summarized 
in Appendix A. 


Error Catching 

The macro is designed to be user friendly, meaning that if users specify impossible 
conditions, the macro will output infonnative messages about what it needs in order to run. For 
example, if the user misspells their count variable or score functions so that the macro is unable 
to find them within the specified dataset, the macro will stop running and output one of the 
following error messages to the SAS log: 
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ERROR: Unable To Find Your Count Variable In Your Dataset. 


ERROR: Unable To Find Your Score Function Variable(s) In Your Dataset. 

Additional messages give specific feedback to the user when the macro cannot locate the 
dataset within the specified library, when the dataset has missing values, or when the user does 
not list a count variable or any score functions at all. 

Limitations 

The macro has been found to produce acceptable results for a variety of smoothing 
problems, but it sometimes fails to converge. When the model contains a large number of 
parameters (e.g., >10 or 12 moments for some univariate problems), SAS IML is less able to 
solve the required linear systems that are necessary for computing the Newton-Raphson update 
for the parameter values. As a result, the macro will terminate and give an error message about 
singular matrices. 

Within SAS IML, the procedure for finding matrices that solve linear systems is 
intentionally limited by the machine’s precision (SAS Institute, 2002b). Even with convergence 
in the overall solution, the SAS macro will sometimes be unable to compute a matrix inverse 
required for the standard errors of p. In this situation, everything except the standard errors of P 
will be produced. Our attempts to work around these constraints considered the use of singular 
value decomposition to compute the required matrix inverses while using a more liberal 
singularity criterion. The results of these attempts were converged but incorrect solutions. We 
therefore treat the SAS-imposed singularity constraint as a necessary balance of the flexible 
Newton-Raphson algorithm (which allows for the fitting of a variety of different kinds of 
parameters) and the storage constraints of the SAS system. One promising possibility for 
improving the convergence rate of the SAS IML macro involves the use of orthogonal 
polynomials of the scores rather than powers of the scores, a possibility that directly resolves the 
singularity issues with B and, as mentioned earlier, allows for comparisons of the Ps. 

Convergence Criteria 

The strictest usable values for convergence criteria should be no smaller than the square 
root of machine precision (Press, Teukolsky, Vetterling, & Flannery, 1992, p. 398). Since SAS 
stores numbers as eight-byte reals, the machine precision is about le‘ 15 . Therefore the strictest 


8 



convergence criteria in SAS would be vie 15 « Se'^. These convergence values may be overly 
strict, especially because of collinearity issues with B. Both criteria are labeled in the macro code 
so that the user may consider larger values for difficult smoothing problems. 

Univariate Smoothing Example 

In this section, a univariate smoothing from von Davier et al. (2004, p. 99-105) is 
reproduced. All of the code and output for this example is provided in the appendices, and the 
reader is invited to follow along with the analyses and also to evaluate the results in terms of the 
original work. The data are already in the form of a score distribution, where the test (7) is a 20- 
item rights-scored test that was taken by 1,455 examinees. Appendix B illustrates how the 
frequency data are entered into a SAS dataset and also how the score functions needed for a 
model that preserves the first three moments of the distribution of Y are defined. The following 
model is fit: 

log, (Pj )= a+ p x (y .)' +/? 2 (y } ) 2 + /? 3 (jv ) 3 (15) 

Appendix C shows SAS macro commands and gives the resulting output. Overall, the 
model fits the data very well, as suggested by the small likelihood ratio chi-square statistic 
(20.24) relative to the degrees of freedom (17). The dataset outputted from the macro (named 
“outresults”) contains the frequencies and score functions in the original dataset plus smoothed 
counts, smoothed probabilities, Freeman-Tukey residuals, and the C-matrix. This dataset is 
shown in Appendix D. Appendix E shows how to obtain the plot of the observed and smoothed 
frequencies. Appendix F shows how the moment-matching characteristics of the smoothed 
results can be verified within SAS. This conversion of datasets of smoothed frequencies into 
datasets of individual scores based on the smoothed frequencies may provide useful inputs into 
other routines that rely on datasets of individual observations, but it also makes expensive time 
demands for large frequency tables. The more direct way of computing distribution moments 
from the probabilities is demonstrated in the trivariate smoothing section. 

Bivariate Smoothing Example 

In this section, the results of a bivariate loglinear problem from von Davier et al. (2004, 
p. 155-167) are reproduced. These data come from the fall 2001 national administration of a 
high-volume testing program. The bivariate distribution is of a total test ( X) with 78 items (J = 0 
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to 78) and an external anchor (A) with 35 items (/ = 0 to 35). The descriptive statistics, based on 
a sample of 10,634 examinees, are included in Appendix G. The tests are correlated at .88, and 
out of the 2,844 possible score combinations, 1,502 have zero frequencies. The 22-parameter 
model to be described next does not converge using SAS PROC GENMOD. 

These data exhibit some unusual patterns that suggest special considerations for the 
loglinear model. First, the two marginal distributions have teeth, a regular pattern of cells with 
frequencies that are much lower than those of neighboring cells. The teeth are due to the use of 
rounded fonnula scores and are at every 5 th score from score 5 on for X and from score 2 on for 
A. Second, there are lumps (very large frequencies) at score 0 in both marginal distributions, due 
to the rounding of all negative scores to zero. Because these patterns are due to aspects of the test 
and the processing of its scores and not to randomness in the sample, they are explicitly 
incorporated into the bivariate loglinear model. 

Under the assumption that the sample bivariate frequencies rijj, have an independent, 
approximate multinomial distribution with population cell probabilities pji, the fitted model is the 
following: 

i°g e C Pji) = a +Px s xo + AAo + £/?**( x j)' + + 2 

1=1 h =1 g= 1 

+YjPxe S Xs( X j) e + Yjfiad S as( a i) d 
e=0 d =0 

The terms of (16) are defined as follows. 

The pji are the population probabilities of obtaining score Xj on test X and score a / on test 
A. The a is a scaling constant that constrains the sum of all of the p /s to 1. The S x o and S a o terms 
are indicator functions set to 1 when xj and a/ are 0 and set to 0 when Xj and ai are not 0. They 
will preserve the lumps at zero in X and A. The four x/ and al‘ terms are univariate score 
functions (the score on tests X and A to the power of 1, 2, 3, and 4) that will preserve the first 
four moments of the marginal distributions of X and A. The four xfaf terms will preserve four 
different degrees of dependence in X and A: XA (the covariance), XA 2 , X 2 A, and X 2 A 2 . The S xs and 
S as terms are indicator functions equal to 1 when x, and a/ are teeth scores and zero otherwise. 

The S X s(Xf ) and S X s ( u i ) terms will preserve the total frequencies of the teeth of X and A when c 1 
d = 0 and the first three moments of the distributions of the teeth (e and d go from 1 to 3). All of 
the Ps are free parameters to be estimated by the model-fitting algorithm. 


f =i 
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von Davier et al (2004) discuss the importance of three of the cross-moments that 
describe the joint distribution, XA 2 , X 2 A, and X 2 A 2 , in the context of equating. In addition to (16), 
we also considered an alternative model, (17), that preserves everything in (16) except for the 
three cross-moments (XA 2 , X 2 A, and X 2 A 2 ). Model (17) is not shown here. Appendix H shows the 
SAS code from importing the dataset to defining the needed score functions for fitting the 
models. Appendix I shows how the SAS macro is called for (16) and (17) and also compares the 
overall fit statistics from both models. 

The model fit statistics for each individual model are not ^-distributed with such sparse 
data, but the use of significance tests for comparing the fit statistics of limited models can still be 
meaningful (Haberman, 1977). The difference between the likelihood ratio chi-square statistics 
for (16) and (17) is 600 on 3 degrees of freedom, significant beyond p < .0001. The two fit 
statistics that penalize for overfitting models (the AIC and CAIC) are also smaller for the more 
complex model in (16). These results provide support for including the three cross-moment terms 
in the model (XA , X A, and X~A~.). Finally, we give the smoothed and observed plots of the 
marginal distributions of X and A in Appendices J and K, where the smoothed frequencies are 
based on (16) (von Davier et al., p. 156-157). These plots show close matches between the 
observed and smoothed distributions, especially with respect to the teeth of the distributions. 

Trivariate Smoothing Example 

Trivariate loglinear models can be used to consider if and how the score distributions of 
examinee subgroups differ from the overall score distributions. This trivariate smoothing 
example extends the 22-parameter bivariate model from the previous section in two ways. First, 
the probabilities to be modeled are defined in terms of the three variables, test A, test A, and a 
subgroup function. Second, the model is extended to consider how the subgroups’ X, A, and XA 
distributions differ. Four trivariate models are considered, ranging from very simple, where the 
subgroups share all of their parameters, to very complex, where the subgroups are considered 
independent and differ in tenns of all of their parameters. 

The Subgroup 

Because the XA exam is a verbal assessment, an important way to subgroup the examinees 
is based on their experience with English. This is captured by examinees’ responses to a question 
about their first language. On the basis of responses to this question, examinees were classified 
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into two groups, one group of 6,867 examinees where English was their first and only language, 
and a second group of 3,767 examinees who were exposed to languages other than English. The 
descriptive statistics on tests A and A for these two sets of examinees are presented in Appendix L 
The XA correlation is .88 for the examinees with English as their first and only language and .89 
for the examinees who were exposed to languages other than English. The statistics reveal 
differences in the marginal distributions for each examinee group. Specifically, the English-only 
students are a better-performing and a slightly more homogeneous group. 


Four Trivariate Models: (18), (19), (20), and (21) 

The first loglinear model fit to the trivariate data is based on the well-fitting bivariate 
model considered in the previous section. To review, this model preserves the first four moments 
in X and A, lumps at zero in tests X and A, the frequencies and first three moments of the teeth of 
tests X and A, and four cross-moments in X and A. To this model we add an indicator function 
(S c ) defined as 1 for the English-only examinees and zero for the other examinees, which 
preserves the frequencies in each subgroup. The resulting 23-parameter model is: 

l°g e ( Pjlc ) = a + Px^XO + Pa Xo + X Pxl ( X j y + X Pa ( U 1 f + X X Pgf (*/ y y 

i=l h =l £'=1 /=! (18) 

e=0 d =0 


Model (18) is nested within two more complex models defined to incorporate specific 
kinds of subgroup differences in the distributions of A, A, and XA. The first of these two models, 
(19), evaluates the extent to which subgroup distributions differ with respect to the marginal 
distributions of A and A. We added four parameters to (18) in order to consider the extent to 
which the fit of (18) can be improved by allowing the means and variances of A and A to differ 
by subgroup in a 27-parameter model: 


'og, (p/ic )= a + p x s xo +p a s M + x Pxi (xj y + x Pah 0/ y + x x pgf (x j y )■ 

g =i /=i 


/=i 


h =1 


(19) 


+2X S »A)' + ZA d s.M)U p e s c . + £a (M) S'(xj)' +XA, M" 


e=0 


d =0 


i =1 


h =1 
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Another model, (20), evaluates the extent to which subgroup distributions differ with 
respect to their bivariate XA distributions: 

tog. e(Pjlc) = a + Px S XO + Pa S aO + £&<(*/)' + YjPaM't + Z Z Pgf ( X j^ (°l / 

*=1 *=1 g=l /=1 

e=0 t/=0 j=l /t=l 

+ ZZA g+ 3)(/ + 3)‘ S 'c(^) g («/> / 

g=l /=1 


Finally, a fourth model, (21), was fit that allowed all of the 22 parameters (excluding the 
subgroup frequencies parameter) in (18) to differ by subgroup (45 parameters): 

tog Xp jlc ) = cc + ft x S X0 + J3 a S a0 + Z P xi 0/ )' + Z Pah ( a i) b + Z Y,Pgf( X ]Y(fllY 

i= 1 h= 1 g=l /=1 

+ifijs„ (xj s m )" + pa .+2 (*, y+£ a„, 4 ) s c («, )* 


e=0 
2 2 


rf =0 


/=l 


*=1 


( 21 ) 


+ Z Z As+3*/+3)^c (h ) 8 ( a / )'^ + PxOc^c^XO + PaOc^c^aO + Z Pxec^cPxs ( X i ) 

g=l /=1 <?=0 

3 




d=0 


This fourth model allowed for a consideration of the fit of a model that regarded the 
subgroups as completely independent with respect to their univariate and bivariate distributions 
and also with respect to their zero-score frequencies and the distributions of their teeth (which 
address the omitting pattern differences of the examinees). Models (18) through (21) allow for 
direct evaluations of how the X and A distributions differ. The SAS code that imports the 
trivariate data and defines the score functions needed for the four models is shown in Appendix 
M. Appendix N shows how the three models are fit using the SAS macro and also presents the 
overall model statistics from the four models. 

Comparisons of the fit statistics across the four models suggest that the subgroups differ 
much more in terms of their marginal X and A distributions than their bivariate XA distributions. 
When the means and variances of A and A are allowed to differ by subgroup in (19), the 
likelihood ratio chi-square measure improves relative to (18) by 114.73 with these additional 4 
degrees of freedom (significant beyond p < .0001). This is almost all of the improvement in the 
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likelihood ratio chi-square statistic that could be obtained by allowing all of the parameters to 
vary by subgroup in (21) (132.10). The AIC and CAIC measures also decrease, suggesting that 
the additional parameters are not overfitting the data. To gain further insight into how the 
marginal distributions of X and A differ, Appendices O and P plot them for the two subgroups 
and also for the total group based on (19). The differences are visible, showing that the English 
examinee group has higher means and smaller variances on X and A. 

When the subgroups are allowed to differ in terms of their bivariate distribution, the 
improvement in model fit is not as dramatic. When considering (20) relative to the simpler model 
in (19), the likelihood ratio chi-square is reduced by 7.66 on 4 degrees of freedom (p > .10). The 
AIC and CAIC statistics actually increase, which suggests that allowing the bivariate XA 
moments to vary by subgroup could be overfilling the very sparse bivariate data. To gain further 
insight into the subgroup differences in the conditional distribution of Xgiven A, the conditional 
means and standard deviations of X for each score of A are computed and plotted in Appendix Q, 
based on (20). These conditional statistics are almost exactly the same for the two groups, with 
the exception at the anchor score of zero, where the English as first language examinees are 
shown to have larger means and standard deviations on X. 

Research Implications of This Study 

The primary objective of this paper was to make a flexible and powerful SAS IML macro 
available for loglinear smoothing. The demonstrations provided in this paper show that many 
different kinds of smoothings can be perfonned with the macro, including univariate, bivariate, 
and trivariate problems. 

A second objective of this paper is to promote the use of loglinear modeling for 
comparing distributions and models, in addition to the smoothing of score frequencies. These 
comparisons of distributions are directly relevant to the consideration of different equating 
models. When loglinear models are used along with the kernel equating framework (von Davier 
et al., 2004), alternative tests of the same question can be considered. For example, the 
comparison of subgroups’ distributions that was featured in this paper’s trivariate section has an 
analogous significance test of group-equating functions at test-score levels within kernel 
equating (through using the discussed trivariate loglinear smoothing results as input and the 
standard error of equating differences). Comparisons of distribution-level and equated-score- 
level results are only now being performed (Moses, Yang, & Wilson, 2005). The approaches 
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have implications for showing the additional noise that equating functions add to score 
distributions, for assessing the practical implications of differences between equating functions, 
and for directly testing the distributional assumptions of specific equating approaches (linear vs. 
curvilinear and population invariance assumptions). 
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Appendix A 

The Steps for Implementing the SAS Macro for Loglinear Smoothing 

1. Enter the dataset into SAS. We assume that most of the time this initial dataset lists 
the test score(s) for each individual examinee. 

2. Convert the dataset from 1) into a frequency dataset that lists the counts for each 
observed test score or observed combination of test scores. 

3. Because not every test score or combination of test scores is always attained in an 
observed sample, an additional, empty frequency table should be created that includes 
all possible scores or score combinations. For bivariate data of total tests and internal 
anchors, this step should exclude structural zeros, the impossible score combinations 
(e.g. if scored as “number rights,” the anchor score cannot be greater than the total 
test score). 

4. Merge the datasets in 2) and 3) and convert any missing counts to zero, 

5. Define all of the score functions needed to preserve the desired moments in the 
smoothing model. 

6. Fit the model with the SAS macro. This step will require only two lines of code, 
which are the following: 

%include 1 [filelocation] \loglinmacro.sas'; 

%loglin (libname=,data=,count=,scoref=,output=); 

The first line calls in the file of the macro from an accessible drive location and only 
needs to be run once at the beginning of the SAS session. The second line runs the macro, where 
library for the dataset can be optionally named after libname=, and a dataset within the library 
that contains the score frequencies is listed after data=, the observed counts to be smoothed are 
listed after count=, and the score functions that correspond to the moments that are to be 
preserved are listed after scoref=. Several measures of model fit will be printed and saved to an 
outputted dataset named f itoutput, where output is specified after output=. A second 
outputted dataset, named after output=, will include the test score values and score functions, 
the observed counts, the smoothed counts, the smoothed probabilities, the Freeman-Tukey 
residuals, and the C-matrix. 
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Appendix B 

Univariate Example: Entering the Frequency Table and Defining the Score Functions 

data llin; 
input count y; 
cards; 

0 0 
4 1 
11 2 

16 3 
18 4 
34 5 
63 6 
89 7 
87 8 
129 9 

124 10 
154 11 

125 12 
131 13 
109 14 
98 15 
89 16 
66 17 
54 18 
37 19 

17 20 
/ 

data llin;set llin; 
y2=y**2; 
y3=y**3; 
run; 
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Appendix C 

Univariate Example: Running the Smoothing Macro and Its Results 


%include 'H:\ApSplreq\tpm\loglinearsmoothing\aera\loglinmacro.sas 1 ; 
%loglin (data=llin,count=count,scoref=y y2 y3,output=outresults); 

RESULTS 

The following score functions were included: 

SCOREFUNCTIONS 

Y Y2 Y3 

ITER 

The solution converged in 6 iterations. 

MODELFIT 


Likelihood Ratio Chi-square 

20.24 

Pearson Chi-square 

18.35 

Freeman-Tukey Chi-square 

20.09 

AIC 

28.24 

CAIC 

53.37 

Degrees of Freedom 

17.00 


BETAESTIMATES 

Betas StdErrors 

Y 0.8389425393320 0.0917352777665 
Y2 -.0453897017972 0.0087387126508 
Y3 0.0005035366392 0.0002577475833 

ALPHA 

-6.748485 
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Appendix D 

Univariate Example: The Dataset Outputted From Running the SAS Smoothing Macro 


proc print data=outresults noobs;run; 


count 

y 

y2 

y3 

smoothedcounts 

smoothedprobs 

ftresiduals 

CY 

cY2 

cY3 

0 

0 

0 

0 

1.706 

0.001173 

-1.79729 

-.000090589 

-.000184074 

-.000285245 

4 

1 

1 

1 

3.775 

0.002594 

0.22371 

-.000183128 

-.000330256 

-.000429494 

11 

2 

4 

8 

7.649 

0.005257 

1.15954 

-.000336071 

-.000527477 

-.000544529 

16 

3 

9 

27 

14.242 

0.009788 

0.50944 

-.000560484 

-.000744582 

-.000545922 

18 

4 

16 

64 

24.436 

0.016794 

-1.33537 

-.000849736 

-.000915463 

-.000345930 

34 

5 

25 

125 

38.752 

0.026634 

-0.74332 

-.001170120 

-.000949703 

0.000093103 

63 

6 

36 

216 

56.978 

0.039160 

0.80741 

-.001459501 

-.000764005 

0.000707042 

89 

7 

49 

343 

77.905 

0.053543 

1.23973 

-.001638764 

-.000324827 

0.001318586 

87 

8 

64 

512 

99.354 

0.068284 

-1.25213 

-.001634929 

0.000318990 

0.001690498 

129 

9 

81 

729 

118.543 

0.081473 

0.96117 

-.001407797 

0.001035082 

0.001628788 

124 

10 

100 

1000 

132.724 

0.091219 

-0.74696 

-.000968368 

0.001644944 

0.001083547 

154 

11 

121 

1331 

139.868 

0.096129 

1.18530 

-.000379933 

0.001985664 

0.000188722 

125 

12 

144 

1728 

139.154 

0.095638 

-1.20858 

0.000259296 

0.001967721 

-.000784962 

131 

13 

169 

2197 

131.097 

0.090101 

0.01328 

0.000844677 

0.001602719 

-.001539598 

109 

14 

196 

2744 

117.307 

0.080623 

-0.75635 

0.001293062 

0.000991380 

-.001864801 

98 

15 

225 

3375 

100.000 

0.068728 

-0.17559 

0.001560260 

0.000281782 

-.001703649 

89 

16 

256 

4096 

81.457 

0.055984 

0.84239 

0.001644003 

-.000380778 

-.001148582 

66 

17 

289 

4913 

63.596 

0.043709 

0.32865 

0.001574774 

-.000892001 

-.000383198 

54 

18 

324 

5832 

47.732 

0.032806 

0.91084 

0.001400550 

-.001204595 

0.000393841 

37 

19 

361 

6859 

34.545 

0.023742 

0.44973 

0.001171820 

-.001323284 

0.001029168 

17 

20 

400 

8000 

24.180 

0.016619 

-1.51966 

0.000930977 

-.001287237 

0.001442615 



Appendix E 

Univariate Example: Plotting the Observed and Smoothed Counts 


axisl label=(angle=-90 rotate=90 'Frequency' font='Times New Roman' 
height=2in); 

axis2 order=(0 to 20 by 1) label=('Score' height=100in ); 

Legendl label=(height=l position=top justify=center '') 

value=('Observed' 'Smoothed') position=(top center); 
symboll color=green interpol=none width=l value=triangle height=l; 
symbo!2 color=blue interpol=none width=l value=circle height=l; 
proc gplot data=outresults; 

plot count*y smoothedcounts*y / overlay vaxis=axisl haxis=axis2 
legend=legendl; 

title 'Figure 1. Observed and Smoothed Frequencies'; 
run;quit; 


Figure 1. Observed and Smoothed Frequenaes 


AAA Observed O O O Smoothed 



Score 
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Appendix F 

Univariate Example: Verifying That the Smoothed Distribution Preserves the First 
Three Moments of the Observed Distribution 

/*These commands create large datasets based on the actual and fitted frequencies of X. 

Then i compare the moments of these datasets.*/ 

data yobserved;set outresults; 

do i=l to 1000*count; 

output; 

end; 

drop i; 

data ysmoothed;set outresults; 

do i=l to 1000*smoothedcounts; 

output; 
bJ 

oh end ; 

drop i; 

data yobssmooth;merge yobserved(rename=y=yobserved) ysmoothed(rename=y=ysmoothed); 
proc means data=yobssmooth mean std skew kurt; 
var yobserved ysmoothed; 

title 'Moments based on the actual and smoothed frequencies of Y. ' ; 
run; 

Moments based on the actual and smoothed frequencies of Y. 

The MEANS Procedure 

Variable Mean Std Dev Skewness Kurtosis 

///////////////////////////////////////////////////////////////////////// 

yobserved 11.5931271 3.9342663 -0.0626866 -0.4988359 

ysmoothed 11.5931404 3.9342451 -0.0626767 -0.4277949 


///////////////////////////////////////////////////////////////////////// 



Appendix G 

Bivariate Example: The Descriptive Statistics of the Marginal Distributions of X and A 

proc means data=neatxa mean std skew kurt min max; 
var x a; 

title 'Table 1: Descriptive statistics of the two tests, X and A (N=10,634). 1 ; 
run; 

Table 1: Descriptive statistics of the two tests, X and A (N=10,634). 

The MEANS Procedure 

Variable Mean Std Dev Skewness Kurtosis Minimum Maximum 

ffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffff 

M X 39.2656573 17.1952746 -0.1014811 -0.7825456 0 78.0000000 

^ A 17.0539778 8.3332670 -0.0096494 -0.8534236 0 35.0000000 


ffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffff 



Appendix H 

Bivariate Example: Implementing Steps 1-5 of the SAS Routine 

/*Step 1: Inputting the dataset of Individual Observations*/ 
data neatxa; infile 'H:\bivariate.dat'; 
input @9X2. 

@11 A 2.; 

/*Step 2: Obtaining the Observed Bivariate Frequencies.*/ 
proc freq data=neatxa noprint;tables X*A / out=neatxa;run; 

/*Step 3: Defining all possible score combinations.*/ 

data outxa,-do x=0 to 78 by l;do a=0 to 35 by 1; output; end; end; 

/*Step 4: Merging 2 and 3*/ 

data neat;merge neatxa outxa;by x a; if count=. then count=0; 

/*Step 5: Defining the Score Functions for the Desired Models.*/ 
data neat;set neat; 

/*The score functions for X.*/ 
x2=x**2; 
x3=x**3; 
x4=x**4; 

/*The score functions for A.*/ 
a2=a**2; 
a3=a**3; 
a4=a**4; 

/*The X-A Cross-Moments.*/ 

ax=a*x; 

a2x=a*a*x; 

ax2=a*x*x; 

a2x2=a*a*x*x; 

/*The lumps at zero for X and A.*/ 
if x=0 then 1X0=1;else 1X0=0; 
if a=0 then IA0=l;else IA0=0; 

/*The teeth of X.*/ 

do i=5 to 75 by 5;if X=i then IXS=l;end; 
if IXS=. then IXS=0; 

/*The teeth of A.*/ 

do j=2 to 32 by 5;if a=j then IAS=l;end; 
if IAS=. then IAS=0; 
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/*The univariate moments for the teeth of 
IXSx=IXS*x; 

IXSx2=IXS*x2; 

IXSx3=IXS*x3; 

IASa=IAS*a; 

IASa2=IAS*a2; 

IASa3=IAS*a3; 


and A.*/ 
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Appendix I 

Bivariate Example: The SAS Code and Output for Fitting (16) and (17) 

/*Model 16*/ 

%loglin (data=neat,count=count,scoref=X X2 X3 X4 A A2 A3 A4 1X0 IAO IXS IXSX 
IXSX2 IXSX3 IAS IASA IASA2 IASA3 ax ax2 a2x a2x2,output=modell6); 

/*Model 17*/ 

%loglin (data=neat,count=count,scoref=X X2 X3 X4 A A2 A3 A4 1X0 IAO IXS IXSX 
IXSX2 IXSX3 IAS IASA IASA2 IASA3 ax,output=modell7); 

data results ,-merge fitmodell6 fitmodell7; 
proc print data=results noobs,-run; 


The SAS System 


STATS 

MODEL16 

MODEL17 

Likelihood Ratio Chi-square 

1966.93 

2566.90 

Pearson Chi-square 

6466.90 

2.4227E14 

Freeman-Tukey Chi-square 

1632.96 

1780.86 

AIC 

2012.93 

2606.90 

CAIC 

2203.18 

2772.33 

Degrees of Freedom 

2821.00 

2824.00 
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Appendix J 

Bivariate Example: Plotting the Observed and Smoothed Counts of X 

proc means data=modell6 noprint;var count smoothedcounts;class x;output 
out=xgraph sum=observed smoothed;run; 
data xgraph,-set xgraph;if x=. then delete; 

axisl order=(0 to 250 by 50) label=(angle=-90 rotate=90 'Frequency' 
font='Times New Roman' height=2in); 

axis2 order=(0 to 78 by 5) label=('Score' height=100in ); 

Legendl label=(height=l position=top justify=center '') 

value=('Observed' 'Smoothed') position=(top center); 
proc gplot data=xgraph; 

plot observed*x='plus' smoothed*x='square' / overlay vaxis=axisl haxis=axis2 
legend^legendl; 

title 'Figure 2. Observed and Smoothed Frequencies'; 
run;quit; 


Figure 2, Observed and Smoothed Frequenaes of X 


AAA Observed O O O Smoothed 
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Appendix K 

Bivariate Example: Plotting the Observed and Smoothed Counts of A 

proc means data=modell6 noprint;var count smoothedcounts; class a,-output 
out=agraph sum=observed smoothed;run; 
data agraph,-set agraph;if a=. then delete; 

axisl order=(0 to 500 by 100) label=(angle=-90 rotate=90 'Frequency' 
font='Times New Roman' height=2in); 

axis2 order=(0 to 35 by 5) label=('Score' height=100in ); 

Legendl label=(height=l position=top justify=center '') 

value=('Observed' 'Smoothed') position=(top center); 
proc gplot data=agraph; 

plot observed*a='plus' smoothed*a='square' / overlay vaxis=axisl haxis=axis2 
legend^legendl; 

title 'Figure 3. Observed and Smoothed Frequencies'; 
run;quit; 


Figure 3. Observed and Smoothed Frequencies of A 


AAA Observed 0 O O Smoothed 
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Appendix L 

Trivariate Example: Descriptive Statistics of the Subgroups’ Marginal Distributions on X and A (Step 1) 

/*Step 1: Inputting the dataset of Individual Observations*/ 
data neatxa; infile 'H:\trivariate.dat'; 
input @7 efl $1. 

@9X2. 

@11 A 2.; 

data neatxa;set neatxa; 
length English $9.; 
if x<0 then x=0; 
if a<0 then a=0; 

if efl='A' then do;ie=l; English='First';end; 
else do;ie=0; English='Not First',-end; 
to 

proc means data=neatxa mean std skew kurt mm max; 
var x a; 
class English; 

title 'Table 2. Comparing the Distributions of X and A for the Subgroups.'; 
run; 

Table 2. Comparing the Distributions of X and A for the Subgroups. 

The MEANS Procedure 

English N Obs Variable Mean Std Dev Minimum Maximum 

////////////////////////////////////////////////////////////////////////////////////////////// 

First 6867 X 40.2018349 16.7402678 0 77.0000000 

A 17.5545362 8.1164723 0 35.0000000 

Not First 3767 X 37.5590656 17.8716481 0 78.0000000 

A 16.1414919 8.6414012 0 35.0000000 


////////////////////////////////////////////////////////////////////////////////////////////// 



Appendix M 

Trivariate Example: The Data Processing Steps (2-5) 


data neatxal,-set neatxa;if ie=l;run; 
data neatxaO,-set neatxa;if ie=0,-run; 

/*Step 2: Obtaining the Observed Bivariate Frequencies.*/ 
proc freq data=neatxal noprint;tables X*A / out=neatxal;run; 
proc freq data=neatxaO noprint;tables X*A / out=neatxaO;run; 

/*Step 3: Defining all possible score combinations.*/ 

data outxa;do x=0 to 78 by l;do a=0 to 35 by 1;output;end;end; 

/*Step 4: Merging 2 and 3*/ 

data neatO,-merge neatxaO outxa;by x a; if count = . then count=0;ie=0; 
data neatl,-merge neatxal outxa;by x a; if count = . then count=0;ie=l; 

data neat;set neatO neatl; 

/*Step 5: Defining the Score Functions for the Desired Models.*/ 
data neat;set neat; 

/*The score functions for X.*/ 
x2=x**2; 
x3=x**3; 
x4=x**4; 

/*The score functions for A.*/ 
a2=a**2; 
a3=a**3; 
a4=a**4; 

/*The X-A Cross-Moments.*/ 

ax=a*x; 

a2x=a*a*x; 

ax2=a*x*x; 

a2x2=a*a*x*x; 

/*The lumps at zero for X and A.*/ 
if x=0 then 1X0=1;else 1X0=0; 
if a=0 then IA0 = l,-else IA0 = 0; 

/*The teeth of X.*/ 

do i=5 to 75 by 5;if X=i then IXS=l;end; 

/*The teeth of A.*/ 

do j=2 to 32 by 5;if a=j then IAS=l;end; 
if IXS=. then IXS=0; 
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if IAS=. then IAS=0; 


/*The univariate moments 
IXSx=IXS*x; 

IXSx2=IXS*x2; 
IXSx3=IXS*x3; 

IASa=IAS*a; 

IASa2=IAS*a2; 
IASa3=IAS*a3; 
run; 


for the teeth of X and A.*/ 


/*Step 5, Subgroups: Defining the Subgroup Functions for the Desired Models.*/ 
data neat;set neat; 

/*The score functions for X.*/ 

iexl=x*ie; 

iex2=x2*ie; 

iex3=x3*ie; 

iex4=x4*ie; 

/*The score functions for A.*/ 

ieal=a*ie; 

iea2=a2*ie; 

iea3=a3*ie; 

iea4=a4*ie; 

/*The X-A Cross-Moments.*/ 

ieax=ie*a*x; 

iea2x=ie*a*a*x; 

ieax2=ie*a*x*x; 

iea2x2=ie*a*a*x*x; 

/*The lumps at zero for X and A.*/ 
if x=0 then 1X0=1;else 1X0=0; 
if a=0 then IA0=l;else IA0=0; 

ieIX0=ie*IX0; 
ieIA0=ie*IA0; 

/*The teeth of X.*/ 

do i=5 to 75 by 5;if X=i then IXS=l;end; 

/*The teeth of A.*/ 

do j =2 to 32 by 5;if a=j then IAS=l;end; 

if IXS=. then IXS=0; 
if IAS=. then IAS=0; 
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ieixs=ie*ixs; 
ieias=ie*ias; 

/*The univariate moments for the teeth of X and A.*/ 
IXSx=IXS*x; 

IXSx2=IXS*x2; 

IXSx3=IXS*x3; 

IASa=IAS*a; 

IASa2=IAS*a2; 

IASa3=IAS*a3; 

ieIXSx=ie*IXS*x ; 
ieIXSx2=ie*IXS*x2; 
ieIXSx3=ie*IXS*x3; 
ieIASa=ie*IAS*a; 
ieIASa2=ie*IAS*a2; 
ieIASa3=ie*IAS*a3; 
run; 
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Appendix N 

Trivariate Example: Fitting the Four Models and Outputting Their Results (Step 6) 


/* Model 18 */ 

%include 'H:\ApSplreq\tpm\loglinearsmoothing\aera\loglinmacro.sas 1 ; 

%loglin (data=neat,count=count,scoref=X X2 X3 X4 A A2 A3 A4 1X0 IAO IXS IXSX IXSX2 IXSX3 IAS IASA IASA2 IASA3 
ax ax2 a2x a2x2 ie,output=modell8); 

/* Model 19 */ 

%loglin (data=neat,count=count,scoref=X X2 X3 X4 A A2 A3 A4 1X0 IAO IXS IXSX IXSX2 IXSX3 IAS IASA IASA2 IASA3 
ax ax2 a2x a2x2 ie ieXl ieX2 ieAl ieA2,output=modell9); 

/* Model 20 */ 

%loglin (data=neat,count=count,scoref=X X2 X3 X4 A A2 A3 A4 1X0 IAO IXS IXSX IXSX2 IXSX3 IAS IASA IASA2 IASA3 
ax ax2 a2x a2x2 ie ieXl ieX2 ieAl ieA2 ieax ieax2 iea2x iea2x2,output=model20); 

^ /* Model 21 */ 

%loglin (data=neat,count=count,scoref=X X2 X3 X4 A A2 A3 A4 1X0 IAO IXS IXSX IXSX2 IXSX3 IAS IASA IASA2 IASA3 
ax ax2 a2x a2x2 ie ieXl ieX2 ieX3 ieX4 ieAl ieA2 ieA3 ieA4 ieIXO ielAO ieIXS ieIXSX ieIXSX2 ieIXSX3 ielAS 
ielASA ieIASA2 ieIASA3 ieax ieax2 iea2x iea2x2,output=model21); 

data results/merge fitmodell8 fitmodell9 fitmodel20 fitmodel21; 
proc print data=results noobs,-run; 


STATS 

MODEL18 

MODEL19 

MODEL20 

MODEL21 

Likelihood Ratio Chi-square 

3624.05 

3514.30 

3502.36 

3491.95 

Pearson Chi-square 

12372.13 

11635.84 

11577.26 

11945.93 

Freeman-Tukey Chi-square 

2932.99 

2826.82 

2812.39 

2802.62 

AIC 

3672.05 

3570.30 

3566.36 

3583.95 

CAIC 

3870.57 

3801.91 

3831.06 

3964.45 

Degrees of Freedom 

5664.00 

5660.00 

5656.00 

5642.00 



Appendix O 

Trivariate Example: Plotting the Marginal Distributions of X 
for the Total Group and the Subgroups 

proc means data=model21 noprint;var smoothedcounts;class x ie;output 
out=xgraph sum=smoothed;run; 

data xgraph;set xgraph;if x=. then delete; 

proc sort data=xgraph;by ie,-run; 

axisl order=(0 to 250 by 50) label=(angle=-90 rotate=90 'Frequency' 
font='Times New Roman' height=2in); 

axis2 order=(0 to 78 by 6) label=('Score' height=100in ); 

Legendl label=(height=l position=top justify=center '') 

value=('Total Group' 'Another Language' 'English Language') 
position=(top center); 

symboll color=green interpol=none width=l value=triangle height=l; 
symbol2 color=blue interpol=none width=l value=circle height=l; 
symbol3 color=red interpol=none width=l value=square height=l; 
proc gplot data=xgraph; 

plot smoothed*x=ie / vaxis=axisl haxis=axis2 legend=legendl; 

title 'Figure 4. Comparing Subgroup and Total Smoothed Frequencies on X' 

run;quit; 

Figure 4. Comparing Subgroup and Total Smoothed Frequencies on X 


A A A Total Group O O O Another Language □ □ □ Engl ish Language 
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Appendix P 

Trivariate Example: Plotting the Marginal Distributions of A 
for the Total Group and the Subgroups 


proc means data=model21 noprint;var smoothedcounts;class a ie;output 
out=agraph sum=smoothed;run; 

data agraph;set agraph;if a=. then delete; 

proc sort data=agraph;by ie,-run; 

axisl order=(0 to 500 by 100) label=(angle=-90 rotate=90 'Frequency' 
font='Times New Roman' height=2in); 

axis2 order=(0 to 35 by 5) label=('Score' height=100in ); 

Legendl label=(height=l position=top justify=center '') 

value=('Total Group' 'Another Language' 'English Language') 
position=(top center); 

symboll color=green interpol=none width=l value=triangle height=l; 
symbol2 color=blue interpol=none width=l value=circle height=l; 
symbol3 color=red interpol=none width=l value=square height=l; 
proc gplot data=agraph; 

plot smoothed*a=ie / vaxis=axisl haxis=axis2 legend=legendl; 

title 'Figure 5. Comparing Subgroup and Total Smoothed Frequencies on A' 

run;quit; 


Figure 5. Comparing Subgroup and Total Smoothed Frequencies on A 


AAA Total Group O O O Another Language □ □ □ Engl ish Language 
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Appendix Q 

Trivariate Example: Plotting the Conditional Moments of X Given A for the Subgroups 


/*Comparing Subgroup Differences on the Conditional Smoothed Moments.*/ 
/*Programming the Moments*/ 

data subgroups_pseO;set model20;if ie=0,-run; 
data subgroups_psel;set model20;if ie=l;run; 

proc means data=subgroups_pseO noprint;var smoothedcounts;class a;output 
out=totalO sum=total;run; 

proc means data=subgroups_psel noprint;var smoothedcounts;class a;output 
out=totall sum=total;run; 

data totalO,-set totalO;if a=. then delete; 
data totall,-set totall;if a=. then delete; 

proc sort data=subgroups_pseO;by a;run; 
proc sort data=subgroups_psel; by a;run; 

data subgroups_pseO;merge subgroups_pseO totalO;by a; 
sp=smoothedcounts/total; 

XSp=X* sp; 

run; 


data subgroups_psel;merge subgroups_psel totall;by a; 
sp=smoothedcounts/total; 

XSp=X* sp; 

run; 

proc means data=subgroups_pseO noprint;var xsp;class a;output out=meanO 
sum=xsmoothedmean;run; 

proc means data=subgroups_psel noprint;var xsp;class a;output out=meanl 
sum=xsmoothedmean;run; 

data meanO,-set meanO;if a=. then delete; 
data meanl;set meanl;if a=. then delete; 

data subgroups_pseO;merge subgroups_pseO meanO;by a; 

x2sp=(x-xsmoothedmean)**2*sp; 

run; 


data subgroups_psel;merge subgroups_psel meanl;by a; 

x2sp=(x-xsmoothedmean)**2*sp; 

run; 
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proc means data=subgroups_pseO noprint;var x2sp,-class a,-output out=variancesO 
sura=xsraoothedvar;run; 

proc means data=subgroups_psel noprint;var x2sp,-class a,-output out=variancesl 
sum=xsmoothedvar;run; 

data variancesO;set variancesO;if a=. then delete; 
data variancesl;set variancesl;if a=. then delete; 

data subgroups_pseO;merge subgroups_pseO variancesO;by a; 
xssd=sqrt(xsmoothedvar); 

data subgroups_psel;merge subgroups_psel variancesl;by a; 
xssd=sqrt(xsmoothedvar); 

proc means data=subgroups_pseO noprint;var xssd;class a;output out=sdO 
mean=xsmoothedsd;run; 

proc means data=subgroups_psel noprint;var xssd;class a;output out=sdl 
mean=xsmoothedsd;run; 

data sdO;set sdO;if a=. then delete; 
data sdl;set sdl;if a=. then delete; 

data xgivenaO;merge meanO sdO;by a;ie=0;run; 
data xgivenal;merge meanl sdl;by a;ie=l;run; 

data xgivena,-set xgivenaO xgivenal; run; 

axisl order=(0 to 80 by 10) label=(angle=-90 rotate=90 'Conditional Means' 
font='Times New Roman' height=2in); 

axis2 order=(0 to 35 by 5) label=('Score' height=100in ); 

Legendl label=(height=l position=top justify=center '') 

value=('Another Language' 'English Language') position=(top center); 
symboll color=blue interpol=none width=l value=circle height=l; 
symbol2 color=red interpol=none width=l value=square height=l;; 
proc gplot data=xgivena; 

plot xsmoothedmean*a=ie /vaxis=axisl haxis=axis2 legend=legendl; 

title 'Figure 6. Comparing the Subgroup Differences in Smoothed E(X|A).'; 

run;quit; 
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Figure 6. Comparing the Subgroup Differences in Smoothed E(X|A) 


OOO Another Language □ □ □ English Language 



Score 


axisl order=(0 to 20 by 5) label=(angle=-90 rotate=90 'Conditional Standard 

Deviations' font='Times New Roman' height=2in); 

axis2 order=(0 to 35 by 5) label=('Score' height=100in ); 

Legendl label=(height=l position=top justify=center '') 

value=('Another Language' 'English Language') position=(top center); 
symboll color=blue interpol=none width=l value=circle height=l; 
symbol2 color=red interpol=none width=l value=square height=l; 
proc gplot data=xgivena; 

plot xsmoothedsd*a=ie /vaxis=axisl haxis=axis2 legend=legendl; 

title 'Figure 7. Comparing the Subgroup Differences in Smoothed SD(X|A).'; 

run;quit; 
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Figure 7. Comparing the Subgroup Differences in Smoothed SD(X|A) 


000 Another Language □ □ □ English Language 
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